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This paper focuses on evaluation questi.on:^ for 
teacher centers, with -the emphasis being that the question is as 
important as the answer. The questioning may be ordered in nine 
points: -(1) For whom are the questions and answers useful? (2) What 
effects are desired? How should teachers change after the program? 
(3) How will this change be observed/described/measured? (U) ihat are 
the success/failure criteria for change? At Mhat level/degree are 
effects expected? (5) What activities and processes are to be used ^in 
effecting change? (6) How will the implementation of selected 
processes be observed/descri'bed/selected? (7) What are the/ 
success/failure criteria for process/activity implementat^ions? (8) 
What is the relationship between processes and change? Is there a 
telationshipj? (9) What do the answers to the preceeding questions 
mean? * What has been learned that gives a direction fjc^r action? When 
these questions have been answered, and a plan formulated .;to reflect 
those answers, evaluation may profitably begin, (MJB) , ' 
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."What is the sound of one hand clapping?" This question or riddle was 

CD 



first posed by the Japanese Zen Master Hakuin (1686-1769) as a mecins. of facili- 
tating enlightenment. "The discipU, given a Koan [riddle] to see through^ was 
encouraged to put his whole strength into the singleminded search for its 

solution, to be ^like a thirsty rat seeking for water^..,* to ca^rry the problem 

* * 1 

with him everywhere, until suddenly, if he were successful, the sol ution/ came. " 

The Koan was a technique originated by the Zen masters to help theitr^shake 
their students out of routine ways of thinkijig and acting; to> open ^p new possi- , 
bilities, and to help the individual student realize his full potential. In 
many ways the master teacher in a teacher center is engaged in these same processes 
helping classroom teachers investigate for themselves different ways of .thinking, 
acting, teaching, and learning. From ny perspective, evaluation is:yet another 
way of serving this same function, for program staff. The evaluation assists 
program staff to stand outside themselves and look at what the^Ve doing; evalu- 
ations can shal:e staff out of routine ways of doing\hings, open up new ppssi- 
bilities, and help staff -members realize their full potential. • ; 

The starting' point in the evaluative process is asking the right questions. 
A basi^c tenet of the Koan educational method is th*at the question is as important 
as the answer . This paper will focus on evaluation questions for teacher centers, 
le answers to these questions. may be as simple^ or as complex, as the ^answer ^ 



^Yoel Hpffman, The Sound of One Hand (New York, 1975i, p. 22. 



-2- 



. to any gi ven Zen Koan , 
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A Flov/er in Bipom * * . ' , 

A monk asked Master .Ummon-; "What rs the pure body of truth?" 
Master Ummon said, ".A'fVower in blorom." 
>J1onk:. "'A flower in bloOTi* - what's it mean?" ' 



Master: 



"Maggots in the shit hole, pus of leprosy, scab over 
a boil.'' 



Without monk apprentices asking "What's' it mean?" there would have been 

, no Zen master in this eighteenth century Oriental version ^f a teacher center 

interaction. "What's it mean?" may be a philos'o'phical , religious, or epistemo- 

logical question. It can. also be the yery concrete, practical question of 

researchers or program *staff poliring'over pages of statistical tables and reams 

of computer print-out 'generated by an evaluation study. 

Consicler a real example from* some of the data collected during the 1973-74 

evaluation of the pilot teacher centers coTiducted for the Office of Education. 

Among educators interviewed in Rhaide Island ^ ^ 

27 percent of* the total sample indicated that they had 

.participated in from one to ei-ght different inservice training ' 
^ sessions sponsored by the RITC. .This included 78 percent of 
the superintendents, 39-,percent of tfie administrators, 
T percent of the secondary school teachers and 22 percent. 
^ of the elementary school teachers sampled. About 40 percent 
.of the respondents reported that;^they received graduate 
credit for their t^'me. The participants generally said that 
, the training met their needs, but 60 percent ijidicated that 
this need still existed after the training\ses$ion".2 

What's it mean? The, answer to that question would seem to depend on who 

is interpreting the data. For some the data may indicate "a flower in bloom;" 

for Others \t may mean "maggots in the shit hole;", and for, Master Ummon it might 

m^an both. , 



2 J 

Robert Covert, John Radzikoski, and Jane aUegel , Evaluating the Four teacher 

Center Pilots: The Second An.nua1 Report , Evaluation Research Center, University 

of Virginia. (.Charlottesville, Va.: 1974), p. 22. , . ^ ^ 



Sharon'^'F^tmar), in commenting' on .the report cited above, takes issue with 
the. conclusion that "inservice training w'as^made more relevant and current" 
based only on evidence t^at "training was offered in a wide variety of innovative 
programs." She questions the "assumption that "new is better" and that "exposure 
brings improvement." She goes on to say th.at: "Without descriptions of actual 
train-rng activities, analyses of what specific needs they addressed, and more 

'precise definitions of their effects, it'"s hard to interpret the findings in 

■3 ■ 

this evaluation." ' ' 

It is^hard for Feiman tQ interpret the data because she has a different 

set of questions than those answered in the cited research. But a simple 

syllogism may make the data 'quite interpretable for others. 

Participation in training sessions is. improvement. 
Jwenty-seven percent participated , in training. 
Twenty-seven percent are improved. * . ^ 

Perhaps this is a poor definition of Improvement — but in whose opinion., 

I showed the data paragraph from the pilot teacher center evaluation to a 

graduate^ student and asked "What's it meanr?" The studen^t replied: ,"40 percent 

of the respondents-got graduate. credit for their time. Tt me^ns taxpayers 

are getting ripped off to raise the credentials of teachers so they'll get higher' ^ 

salaries — under the guise of teaclv|r improvement." , ' 

This long discourse on the perception of flowers versus maggots is not 

.aimed at Timply making th» point that different people perceive things differently.' 

I take that to be ^ tfnjism. The point is that this^truism is regularly and colK. 

sistently ignored in the design of evaluation studies. "Laboring under the jss-um/- 

tions of positivistic social science, we pretenct that'th'ene is some body of data' - 

out tl>ere that has only to be collected at which point v/e will know vhat it all 

means, whether or not it works, whether or nbt we're successful, whether or not 



' - ' 3 ^ ^ 

- SharonoFeiman, "Evaluating Teacher Centers," School Review (May, 1977),' 
In press.' , • . . 
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we're-^ffective. Such data simply do net^exist outside the context of a specific 
group of people wi-th a particular perspective. 

The Implications for evaluation of this truism about differential perceptions 

% f 

are enormous. It means that instead of beginning with the tre^ditional first • , 
question in evaluation textbooks — "What are the program's goals?" — we must 
begin by asking: " Whose goals for the' program will be evaluated?". Instead of/ 
-asking "What^s^it mean?" we must ask'"Who is going to decide what it means?" ^ 



Before exploring the* impl ications of t 
pdint a different way. 



lis perspective let me, try to make the 



Hunting Bears 


y Zen Koans are one educational technique for helping the learner look at 

the world in a different way. Another technique that has emerged over time 

' through journeys into. and out of a variety of rich cultural traditions is the 
Sufi story. Sufi stories, particularly stories about the adven'Jures and follies 
of the incomparable Mulla (Master) rfasrudin, are iu means of comipuni eating ancient . 

' wisdom. "Nasrudin is the classical figure devised by the dervisihes partly for 

. the purpose of halting for a moment" si tuations in. which certain States of mind 

1 

^e inade clear/-... Since Sufism is something which is lived as well, as something 

• i 

which is perceived, a Nasrudin tale cannot in itself produce compel ete enlighten- 
ment.. On the other hand, it bridges the gap between mundane lifeland'k trans- 
mutation of consciousness in a manner which no other literary forni yet produced 

4 ' 1 ° 

. has been able to attain." What, then, can we learn Srom NasrudiYi. about eva-lu- 

\ stiot\ of teacher centers? Let's try the parable about hunting bears'- ' ^ 

A king who enjoyed Nasrrudin's company,, and also Itked to 
.hunt, commanded him to accompany him on a bear hunt. Nasrudin 
was terrified. ■ . ' 

When NasruQin returned to his village, someone asked himi 
; ■ "Hovf did the hunt go?" 

.__ ■ ■ ' ] 



Q ^Idries Sh'ah. 'The- Sufis (New York: 1964), p. 56.. 
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"Marvelous ly;^ ' " . < * 

"How many bears did yeu see?" 
"^one." ^ 

\ ' ' ' • 

• \ ' 

"H9W could it hsLve gone marvelously, then?" ^ . • 

"When you^^ay«e hunting bears, and you are me, seeyi-g no bears 
- at all js^ a marvejous experience."^ • 

Permit me. to translate this tale Tnto modern language. The evaluation, 
report on the king's hunt might read something Tike 'this: 

, ■ - EVALUATION^F THE BEAR PROJECT . •• ' 

" ,' ' • • 

\ This is a study undertaken for His Majesty's Ministry of the Interior, 

under the auspices of the Department of Natural Resources, for the Division of 

Parks, Section on Hunting, Office of Bears. This .is a study of''the relationship 

between the number of bears si.ghted on a' hunt and the number of bears shot Qn 

a hunt. Our hypothesis' is that there is a direct, linear relationship betv^een 

.th*e sighting of bears and. k^l 1 ing^f bears. The data'was collected on a recent 

royal hunting expedition. The sample size is therefore sonfewtat small and . 

generalizations cannot be. made with cbnfidenc"e. In. effect tliis is an exploratory 

case study , Campbelf and Stanley Research Design No . ^1. 

The data- support the hypothesis at the .001 level of sigpificance. Indeed, 

the correlation is perfect. The number of bears 5.ighted was kero, arid the 

number- of bears killed was zero. * In ho case was a bear killed without first 

being sighted. We therefore recommend that in future projects new Royal, regulations 

H. ' ' ^ * I * 

be implemented requiring' that bears first be sighted before they are* killed. 

. ^ - ■ , , i V 

Respectfully submitted, 
* J • 'The Incomparable Mull a Nasrudin 

/ ' * * Royal Evaluator * . 

[, ~ , • ' • 

/ r : ■ — : « . /. 
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Ibid., p. 61. " ■ ' . / " . . ' ' .4 



^- While this evaluation report maV be stati stically somewhat 1p<;s. ri gnrniK 



than the average evaluation study, it shar|s'one major characteristic with 
almorst all other reports of this genre; -It is impossible to tell whefher-or 

• " , if 

not 1t answers anyone's question. Who"'decided that the outcome should be the » 
.number of bears k'illed? Perhaps the pn^'ect skff simply uses the hunt as a 
format for gett\'ng royal (Federal) money to "conduct field trips and the real 
outcome is a> hei ghtened'sensi ti vity ta nature? or a cHoser relationship between, 
Nasrudin and the king? \or#^reduction df.Nasrudin'V Ve'ar of bears? or an'^increase 



.'in the king's paver over Nasrudin? It may even be poss'ible (likely.') that dif- 



ferent characters in the situation have different objectives and would like 
different outcome .n]easu'res . If so, it*seems. unlikely that all characters will 
be interested in the same eval uation -data. .Who will decide what it all )neans?' 
For Nasrudin the data indicated a "marvelous" outcome. Other decision-makers ' . 
might read the data differently. ■ ^ ' " % 

^ • * • • 

Utilization of Evaluation Research * ^ . ' 

I This emphasis on determining who the evaluation will serve as the first 
^tep ig the evaluation process derives from a personal bias that evaluations 
ought to be^ useful tbey ought to have an impact on wfijt people think, plan, 
and dof In 1973, this bias ,led me to -get involved in a-^study of the factors - * ' 
that affect the utilization of evaluation research/ We studied the Ij^ilization 
oT twenty Federal health eval uations- by interviewing the principle program person, 
•evaluator, and project officer' in each case.^ In attempting to identify the -. 
factors that affect utilization of evaluation findi|gs ^e asked respondents to 
comment on the relevance and importance of eleven factors extracted from the , j" 
'literature on utilization: • . * ' 



►"The study, is described in detail in Michael Patton, et. al. , "In «S'earch 
o| Impact: An Analysis of the Utilization of Federal Health Evaluation Research," 
in Using Social Science Research in Pol icy-Making , Carol Weiss; ed, (New Yefrk, 
1977},..pp. 141-163. • 7 

: . • ■ • . ' 7 - . . 
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. — methodological quaO.ity, methodologfcal appr*oprfateness 

timeliness, lateness of ref)ort. -^positive-negative findings, 
surprise of findings', central -peripheral program objectives 
evaluated, presence-absence -of raT^ted studies , political 
factors, goyernment-aevaluator interactions, and resources . ■ 

■ available for the study: • * • 

Fihally, we- asked r^espon-^lepts to "pick out the single factor you fee-1 

had the greatest effeot on how this stud^was use-d." ;^ ' 

Two, related factors emerged as important in our intervie\-/s : (1) a political' 

considerations factor and (^) a factor we have called the personal factor. This - 
. . . , " .. I . ^ • 

. latter factor Was unexpected and .it?, clear importStrice to our respondents has 

substantial implications fon*th6 utilization of evaljjation research. None of - 

the other specific literature factors about which we asked questions emerged 

as important with any consistency . - . ^ 

. The personal factor emerged most dramati<fally in our interviews when, 

/having asked respondents to comment on the importance of each of our eleven 

utilization factors, we asked them to* idmify the single factor that was most 

important in explairiing the impact or lack of impact of that particular study. 

Time 4fter time, the ''factor they identified y/as, not on our list. Rather, they 

responded in terms of the importance Ch^ individual people*. . • 

For lack of a-better term, we have simply calle.d^this ne\^/ variable the personal 

• factor. It is made up of equal P3^ts of leadership,' interest, enthusiasm, 
.determination, commitment, aggreiisi.venes's , and caring.* Where the persdnal factor 
emerges, eVSiluations'have an iij^act; where it is absent, there is a marked absence 

• of impact. ^ 

Comments from one study with an unusually high level of , utilization are 
illustrative. . The decisionmaker was asked why. this particular study had so 
much fmpact. His answer was brief and to the point: 



Well, [the evaluation had an impact] because we designed the 
project with an evaluation conpopent in i-t, so we' 'were expected 
to use it and we did.*. ..Not just the, fact that [.e\^uation] 
jfas^built in, but the fact that we built it in on purpose,, ; 
S • That is, the agency head *and,, myself had broad responsibiTities 

ERJC ^ ■ ■.•/•"•■■ 8- ■ • ' ' 



for this, wanted the evaluation study results and we expected 
•to use them. Therefore they were used. That's my point, jjf 
gcfmeone else had built it in because they thought it ^as needed , 
• • and vue didn't care, I'm sure the use of the study results would 
' have, been di ffer^nt . ^ ' ' 

The evaluator, (arf external agent selected^through an open RFf^ process), 

•completely agreed that: 

* ' ♦ . • * 

The principal reason (for utilization] was because ^the decisionmaker 
; -was' the guy who requested the evaluation and who used -its , results'. 
\ That is, the organizational distance between the pal icy maker and 
the evaluator was 'almost zero in this instance. That's the .most 
. important reason 4t had in impaxit.^ • , . 4 * 

j^hat emerges here is a picture of a decisionmaker who knew v/hat information 

he wanted, an evaluator committed to answering the decisionmaker's question, and 

a decisionmakei^^ committed to using that information. The result was a high level 

of utilization in making a decision contrary/£o^he decisionmaker's initial 

personal hopes. ■ ' , . , . » ' 

• • / 

This point was made often in- the interviews.. *Ohe Kighjy *|d laced and highly \ 

experienced administrator from yet a different project' offered the following 

a<5vice at the end of a fOur hour interview: 

' Win over the program people. Make sure you're-hooked into the 
person who's going to mafee the decision in six months from the 
*• time you're doing the study, and make sure that he feels it's ' 

' his' study, that these are his ideas, and that it's focused ' / 

on his values ^ ... 

The 'message here is in two parts. .First, evalqators cannot conduct useful 

studies unless they know whose question they are ariswering; and .secondly, progv^am 

people have a right and an obTigatiqn to participate in the entire evaluative 

process from design to interpretation if results are to be.usefCil. Mn some cases ^ 

such participation means making demands that are unusual and beyond yie ejcperiences 

of most evaluators. In 1972 the teacher trairting program at the New School for 

Behavioral Studies in Education, University of North Dakota, was to be evaluated 



7|bid., pi 157. 
8lMd.. p. 158. 
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as part of a national' Offi ce of Education study. Dean Vito Perron*e argued that 
the study, as designed, would-be useless to th.e New School. He talked the 
Office of Education people into allowing him to spend ^the Ne\^ School's portion 
of the evaluation money, on a. study locally^ designed and locally conducted. 
The subsequent evaluation was entirely staff designed and produced instruments 
an<(.data that have become an integral part of the North Dakota program'. The • 
national -study produced large- vol ucies of numbers (with'blank^ entered on the 
lines for North Dakota), and as far as I can telT, was of no ^Darticular use to 
anyone/ 

. *• Contemplate, then, the usefulness of evaluation research. Begin the evalu- 
ative process by asking:*- Who is this study for? f}hos;e question will be answered?; 
Begin by contemplating .the usefulness af what to whom. And if you can't answer 
that initial question in terms of real people with real questions, then leave' 
the evaluation and contemplate a Zen Koan; fbr example, the Koan' on snow: "Where 
do all these lovely snowflake's fall, anyway? , Do they" fall in any particular place?" 



, * ^ PROGRESS VERSUS CHANGE- v - • " 

One of- the purposes of evaluation research, for some the major 'purpose, 
is to tell us whetf>er or not we're getting where we want to be. In the preceding 
pages I've sugges-ted that different people involved in 'the same program may want^ 

to get to different places^ Various people v(ilT interpret data about where we 

•j • . ' 

are in different ways. For many the issue centers on the question of improv;ement > , 

Are things improving? 'Are teachers Improving? children doing better? These 

may well be the central questions in an evaluat^ of teacher ^centers . Kathleerr 

Devaney in a paper on Teacher Centers a^rgues that assessment of program effective-, 

ness "will have to b€ in terms of verifying Individual tethers' improvement in 

classroom performance over time "^-^ ' , ' 



O « ''^Kathleen Devaney, "Surveying Teachers' Centers.," Teachers* Centers Exchange , 
ERJC Occasional Paper No. 1, National Institute of Education, (April, 1977), p. 7; italics 
— added, , • • 10 ^ ' \ 



• • < ■ . • . ' 

Let me su jjgest t hat for teacher center evaluations it may be important to 

separate the issue of improvement from the related, but quite different, issue- 

') ■ 

of 'impact or cbAage. Improvement involves a judgment about whether or not 
something is better. * Impact involves the more limited question of whether or ^ * 
^not something is different. An observed difference may or may not constitute 
improvement depending on f^o is making the value judgment about whether *r not a 
change is for better or worse. It is crucial throughout the evaluation process- 
that empirical observations about program impact be kept separate from judgments 
about whether or -not such impact constitutes improvement. 

Suppose -a teacher center conducts a series of workshops on the use of 
resources outside the classroom. As a result, a group 'of teachers increase the 
time students spend \^ the copimunity "by 'an average of three hours a week. The 
time spent oifts.ide the classroom led to an average reduction of one hour per 
weeK.in time spenV in class- work doing both reading and arithmetic. Outsi'^e the^ 
classroom the students are exposed to a .variety of activities arrd opportunities j 
but the'r.e is a net reduction in the time spent doing supervised reading and 
anithmetic. Clearly the teacher center has had an impact. . Change has , occurred, 
•fiut have the teachers "improved"? The answer to that question depends bn how 
much* one val^ies supervised reading and arithmetic compared to other stimulating 

activities. ^ ' * ' ' ' 

^ ' . . . . • 

A Siifi stiry illustrates quite nicely the problem of determining whether or 

not ifnprovement has occurred. The setting is a teahouse, a Sufi term ^for a meeting 

p^ce of dervishes. 'A monk entiers and state?: • % 

"^1y master taugfyt me to spread the^word that mankind will 
never be* fulfil led, until the man who has not befen wronged is as 
indignant about a wrong as the man who actually has been wronged." 



The* assembly is 



momentarily impressed, ^Then Nasrudin speaks : 



"My master taught me that, nobody at all should become indignant 
about anything until he is sure that what he thinks .is a wrong is 



1 1 * ♦ ** 

in facW a wrong — ^.and not a blessing in disgUisel 

* * ' 

Questions of right^and wrong, better or worse, are not simple empirical 

questions. To formulate- evaluatioa questions 'solely in such terms can sabotage 

an evaluation from the beginning. What then can one do? In my judgment* tRe - 

empjrical questiofi>is not- improvement but change. I suggest that, we begin ' not 

witi) the question of whether or not teachers are "better," but whether'or not 

they are different . Has the program been effective' in changing teachers? Do thej| ^ 

^think differently?^ Can they do things no\^/'that they couldn' t .do^before? .99 they* 

feel differently? Are di fferent things occurring in teachers/ classrooms? These 

*■ " ' 

are empirical evaluation questions. Da-ta from these evaluatioli questions can' then 

/^) ' • , ' 

be used to determine whether or not such changes and differences cons^ tut€ jDrogress 

or improvement. ^ ' ' ' 

This is not an esoter-ic, semantic dtstrncti.on. Nor is it the beginning af 

a polemic on value-free social science. It ts a practical suggestion for distin- 

guishing between th^rt which can be observed (by whatever methods) and that which 

caqnot tie observed. Failure to make that^dis'tinction canr lead to-s'erious misuncjer- • 

standings "and mis communications *thrQughout th^ evaluation process- . 

< What, 'then, can be observed?. There a>e essentially five categories of 

changes *in» teachers.^one- might investigate: < 

!• Changes in teachers' feelings '{f.e.\ m^irale,- *suppo>t, / . , 
enthusiasni, isolation, etc:)f 

2. - Changes in teJichers' opiniQns' (i .e, , Interpretations , 
• attitudes, expectations, goals, ^ etc. ) . : ' 
» • • * - " ^ ; • ... - 

• ' : 3. Changes in teachers* k;iowledge (i.e., fa.cts, da£a, theories, 

models', etc/);' * * ^ . . ^ t: ^ ^ , 

4.^ Changes'' in^ teachers* sl^ll? (i.e., things' they ^an do)"; and. 

51 . Changes in teachers ^^ehavior (i.e., things teachers actually., 
' • V . ^ do do in teaching). ^ . ' ' ' . ' ' 



^^Shah, 0£- crt. , pp. ^58-9. * - , . ^. ' " , > . ^ • 

I Thes*e?iYe categories, corresponcj fo the five elements; of awareness* identified 
by Elam NunnaUy, Sherrod ttlller, and ^^)^ Wackman. Alive and. Aware. • • 



A particular teacher cer^ter might focus on any or all of these types .of 
Changes. Prograjns typically vary considerably in degree of emphasis on .different 
outcomes. A comprehensive evaluation might well look at all of these outcomes. ^ 
For^some educators "improvement" only occurs if teacher behavior changes; for 
others changes in teachers* feelings constitute sufficient improvement to justify 
• a prot|ram.. Many educators relate changes together in. a sequence: knowledge 
changes (new information) then attitude change (new understandings) then skill 
change (new capacity) then behavior change (new actions). Where in this chain 
does real improvement begin? Kathleen Devaney quotes one educator to the effect 

that: . I ' , • H/--^. . / 

The key to opening people to change is^ attituder^:iot infor^ 

mation and skills. I can distill into ten pages what I 

think a teacher needs .{in order] to teach reading. But 

before that, teachers musf^get a set of basic^attitudes ^ 

hpw langiiage is viewed, how reading is viewed. Attitudes are 

the crucial foundation that makes it possible to assimilate 'new skills. 

By carefully specifying the changes tha.t staff want to observe in tea-chers 

as a result of teacher center activities, -program staff can clarify and make 

explicit what they mean by "improvement."' The next task is to determine what 

^different levels cf change indicate about the' program If fifty-five percent 

, of the teachers in a workshop actually use a set of materials in their classroom, 

is t^t high usage or low usage? Does that mean tfte workshop was effeptij/e or* 

ineffective? The workshop had. an impact, but what' level of impact is desi/-able? 

What leveh-is acceptable? And'what level spells "trouble? These issues should Be - 



resolved before fR^^^ta is collected to perinit 'the discusston about" what , 
constitutes -improvement to take place in an atmosphere that is not .charged with ^ 
:defensiveness, rationalization, and justification. 

Suppose you are collecting data on frequericy of individual visits as Fei-man . 
did. Her data can be grouped as follows: ^ ' • 



^^0£. Cit., p.. 7. 



ERIC ^Sharon Feitnan, Teacher Curriculum Work Gehter: A Descriptive Study , North 

* ^Dakota Study Group on Evaluation Monograph, University of NortK Dakota Csrand Forks, 



Number of Visits^ . 
Made by a Teacher 
CO trie Lentsr 


Number of 
Vi s 1 ts - 


f 

% of TQtal 
Visitors 


* 


. 1 - 2 . •• 


' 1 85 


80.4% 




3 or more 


- 45 

f ' 1 


19.6% 

L- 


] 



Note: Data are for visits betv^een JanuaVy 10 and February 28. 



• I don't know how st^ff reacted to- these findings • My guess is that once 
tbese data are collected the program staff might .look at them and say, "Oh, 
ye^, trt|t*s about what we We^^ anticipated. Plus the data don't include people 
^who come to regular workshops and speci-al classes. Then, too, 'since only 
23* teachers 'noted on the background forms that they first visited the Center 
during the period of ^observation , it i$ not likely that most Of the people who 
^came once were first-time visitors-.' And the observation time was really too 
short. Plus January and February and bad months, you know, everyone is depressed 
with winter, and. . * ^ ^ 

Soon it becomes apparent that either th^ data don't tell us much, at least 
not without oth^r data, or that staff aren't prepared to-deal with what the 
^tiata do suggest. This is not at all unusual as a post-evaluation scenario. 

Now let OS try another scenario. ^At the outset of the evaluation study^the 

prQg*r*am staff discuss their notions ^of what their task is' and how teacher change 

accurs. The staff decide that the kind of impact they want to have ^annot occur 

in one orJ:w6 visits, to the^teacher center. "If teachers don't return after 

♦ • • 

one or two visits we must be doing something wrong." The period .of time in 
question is a full twelve mofttIT periods Befor e the data ar« collected the staff 
complete the following table:' * - ' J ' 



Interpretation of data 


0f 

Percentage of teachers' 
who have 'contact with the 
Center three or more times 


.We're doing an outstanding - 
job of engaging teachers 
. at tijis level 


51 - 75%' - 


We* re doing an adequate job 
of engaging, teachers at fhis 
level 


_ ^ — 

i.. 


We*r)e doing a poor job of 
engaging teachers at this 
level 


■ ..0-25% 



A record keeping system must then be established that staff agree to and' 
believe in so that the 'data have credibility. The teacher center staff have 
•coirmitted themselves to actively enga^ging teachers gn a multiple, contact basis. 
The data will provide ,cl ear feedback about the effectiveness of the program. 
Th6 key point 'is that if staff are unwilling and/or unable to interpret data 
and set expectancy levels b.efore the evaluation" there is no reason to believe 
they can do so after the evaluation. In addition, going through ^this process 
ahead of time alerts participants to additional *data they need in order to make 
sense of the, evaluation ; clearly one table on freqyency af visits "is only a> 
starting place* Finally, involving staff tn such a process helps clarify the 
evaluation criteria that are being used arid,, if staff are involved in establishing 
these criteria 'themselves , the eval'tiative process will increase their cdhimitment 
ta use the data for program improvement.^ 

Jf multiple contacts with teachei^s is a key issue for center staff they yould 
probably want to conduct follow-- up interviews with teachers who, didn't return and 
teachers who did return, what was the difference? Thera.,are many possibilities 
for extending thi? exarfiple. The point Is^tha^t the' evaluation question must be 
caY-efully fopnulated in accordinace wi th the basic goals'of the teacher center ; 
stafj arg involved in establishing explicit criteria for, interpreting^the data ; 
and staff cbi?ihiit themselves to taking tKe process seriously. 



Nfany of the most serious conflicts in evaluation research are rooted in 
the failure to clearly specffy criteriajn advcfhce of data collectibn.' This can 
lead both to collection of the wrong data, ^nd/or intense disagrbemsnt about the 
standards for interpreting data that have already been collected. Without explicit 
criteria data can be interpreted to. mean almost^ahy^ing -about tlie program — 
and about the quality of teacher center sta1;f. Witness,, for example. Master 
Obafku's fainous evaluation of the ^taff in eighteenth century/ Chinese teacher 
centers: 



MaslEr Obaku said, "You are all^ leftover eatersl If you vfalk 

arfiUnd the world and search for truth in such ^manner, what 
achievement can you expect*? Do you know that there are no • 
more Zen masters in China?" A 

Then a monk stepped out and said, "Aren't there thoseiwho 
walk around earnestly ^instructing ^-he masses? What of 
them?" * - 

Obaku said," "I did not say there is no Zen anymore, only that 
there are no great masters." ^ 



The Outcomes' of Individualization » ^ 

Feiifian reports that most teacher center programs advocate individualization 
in thfit ^'teachers must be allowed to begin at \.\\^ own beginnings, draw on 
personal strengths, and learn at thfeir own pace."^^ This creates special problems 




in evaluating the outcomes of teacher center experiences, particularly problems 

■v. 

interpreting varying levels of change to make interpretations about improvement. 
Measuring improvement for evaluation purposes turns out to be a very tricky 
problem.' The firsfproblem is determining the level at which learners ,enter the • 
situation. Then, for any given knowledge area or skill^ ther^e are upper limits 



of competency. One^can't show improvement indifinitely . It is well-established 
that learninl'll^urs most rapidly and in greatest amount for a person who is just 
beginning to study a new subject. The reason is simply that there is more to 

V 
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learn at the b.eginning/ After a time, percentage gains •decline. Persons v?ho 
have never driven a car can^improve their skills 100%; persons who have driven 
for years may pnly be able to •show a small percentage gain after further instruc- 



tion. How can , the large pem:entage gains in a program of novices be compared 
to the small perjcentage gains in a prograjn of experts? And teacher centers include 
bothi ' ' ^ , ' ' 

An individualized teacher center program requires an individualized set of 
, ' evaluation change criteria that take into consideration where a teacher begins, 
what the teacher wants and needs, and what changes occur after the experience. 
Thfs^may.mean a careful system of descriptive" records for teacher center partic- 
ipants (or a. sample thereof) not unlike the record-keeping 'systems .developed by 
/ ' ^at Qareny^^ for observing- changes in students over time. Such a system permTts 
the recording -of some types of information about each teacher but also permits 
the collection of systematic data on the individual developement of each teacher. 
Such records serve program advisors for* both planning and evaluation purposes. 

The point here is that an evaluation systfem is worthless if it collects data 
about the wrong things . Observing changes in a highly individualized program 

' • - • ' \ 

means monitoring' and describing different changes for different individuals and • 
then looking for overall patterns of teacher change and learner activity in 
the program. ^ " • . 

One evaluation report on a' teacher workshop program tried to solve the problejif 
by simply asking teachers hpw many felt they had changed as a result of their work- 
shop experience. Severity-six percent said they ^ad"changed. But no data was . . 
collected on how they changed or what activities Ted to the' change. In terms 
^ of actual program planning and improvement the data were^n't very useful. 

An oft-repeated Sufi story illustrates* the importance of observing and collectin'j 

'^data on the right thing. \ . * « ' . . 

* ♦ * 

— >4' ^^Pat Careny; A Phenomenological Approach to the Study of Human Phenomena. • -"^ 
PpYr M) ^^^'^^ Dakota Study tiroup on tvaiuation, umversuy or. North uaKota, (brana horks*, 
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, Nasrudin used to. take his donkey across a frontier every 
day, with the pannier^ loaded with straw. Since he* adntitted 
to' being a smuggler when he trudged home every night, the 
frontier guards searched him again and again.- Jhey searched 
his person, sifted th^ straw, stepped it in water,, even 
burned it froiti time tp time. Meanwhile he was becoming vis- 
\ibly more *and more prosperous. 

Theri^he retired and went to live In another country. 
Here one of the customs officers met him, years later. 

"You can tell me now, Nasrudin,*' he. said. "Whatever vvas 
-it that you were smuggling, v/hen we could neverycatch you out?" 



"Donkeys,"" said -Nasrudin. 



/ 




The One-True WajN v ^ . 

, Enlightenment is the goal in Buddhism. Koans and Suff stories are only 
;^two of many technicfues for assisting the learner along the path- toward Enlightan- 
,\»nerit, ^"the shattering of the Great Doubt.,., the weTling up of a flood of exalta- 
tl^on. ^ 'If you take up one Koan,' Master Hakuin said, "and investigate it unceas- 
' * ingly, j^p^ minjl will die and your will will be, destroyed. ' It is as though a 

vasK empty abyss lay before you, with no place to set your hands and feet. 
\/{pu face death and your bosom feels as though it were fire. Then suddenly you 

•are. one with the Koan, and body and mind are cast off You must push forward 

^.entles^ly and with the help of tfiis complete concentration you will penetrate 
without fail to the basic source of your own nature. "^-^ 

The Koans were -a .method 'for attaining Enlightenment, but they were also 
an evaluation tool for detecting Enlightenment. The^e are correct resp.onses 
and incorrect responses to this Master's Koanic question, "What is. the sound of 
one han(4 cjapping?" There are close to two thousand^ recorded Koans ^ — complete^ 
'Sequences of questions -and enlightenedirespons^s. ^ Koans are supposed to be 
passed oa 1n secret fror^Mas^t^er to di^sciple, ♦^'transmitted only by word of mouth 
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to the student in the master*s roomJ^ The pupil vows to keep the Koans a secret 
and to transmit them only to chosen- disciples ^after he has become a Zen Master* ' 

Koans were first published under a pseudonym by a rebel Zen master v/ho had 
become disenchanted with his fellow masters. ' The publication created an enormous 
scandal because once the Koans were made public anyone could supposedly memorize 
the correct responses. Therefore, anyone could become a Zen rriaster. Hw, then, 
was one to tell the truly enlightened from the fakes? Meanwhile, the rebel master 
maintained that monks were already mindlessly repeating Koans without being truly 
enlightened.^ - , . . 

The Koans themselves emerged out of^a debate over whether enlightenment 
came from passive meditation or a more active searching with the mind. Othejr 
Zen monastic retreats disagree over whether Enlightenment conges slowly over ^time 
or quickly, all-at-once. Of such issues are sects created, each teaching the 
One True Way. • ' 

Western social scientists would scoff at such religious absol utism. and, 
under the enlightenment provided by cultural relativism, would explain in great 
depth why each culture deceives itself.with its ov/n version of Truth. ^ These 
same social scientists will then design an educational experijrent to determine 
the one best way to teach students that there is no one best way to attain 
Enlightenment. ' * 

When I finished reading Sharon Feiman's excellent paper on the three models 
of teacher centers'^* I immediately had a vision of a, researcher writing a proposal 
which would be enthusiastically funded by the National Institute of Education to 
determine which of the'tliree models was most effective in changing teachers. 
This despite the fact that every large scale experiment of this kind (witness the 



1 '^I. Miura'and R.F. Sasaki, The Zen Koan (Ryoto: First Zen Institute of 
America, 1965), p. 28. ► 

; •20Hoffmsn, 0£.'Cit. , pp. 39-44. .• ' • ' " 

' ^Veimari (1977), .Op. Cit. • ' ^„ 
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National folloWi Through Eve^Tuatiora) has found that the within-group variationi; 
is greater than | the' betneen-group variation, i.e.,^the "experiment" fails to 
identify a singllfe best "treatme.nt. ? 

But the search goes- on* , Educational programs are compared to each other 
.using standardize^d criteria despite the fact that different pirograinfe are trying 
to do different trrings/- Such evaluation comparisons are particularly insidious 



me 



' because they subtly divert* program staff from their original ^oals* Let . _^ ^ 
^illustna-te the pYoolemajsing the Feiman paper. » | 

^ ' Feiman describes three types of teacher centers: "behavibral" centers, 
"humanistic" centers, and "develepmental " centers\ I apologize for, the injustice 
done to her paper by the table below, but this table represents a simplified 
summary oVrny understanding o-*" the di fferences ' among th^^^jyir^^ models. 



Type of Center 



Primary Process of 
Affecting Teachers 



3 Primary Outcomes of' 
the Process / 



1. Behavioral Centers 



2. Humanistic Centers 



3. Developmental Centers 



Curriculum specialists direct- 
ly and formally instruct ad- 
ministrators and teachers 

Informal, non-directed 
teacher exploration; "teachers 
select their own treatment." 



Advisors establish warm, 
interperson^il , and directive 
rel ationship wi th teachers 
working with them over time\^ 



Adoption of comprehensive 
curriculum systems, methods 
and packages by teachers 

Teadier feel supported and 
important; ptck up concrete 
and pljgtical ideas and 
materials for inmediate use 
in fheir classrooms. 
«k 

Teachers* thinking about wh 
ihey do and why th^y do it 
is changed over time; 
teacher personal developmen 



Regardless qf Whether or not one agrees with Fp'man's categories, it is 
clear tha't, at least to some extent, different teacjier centers are trying to 
accomplish different outcomes . The three (nodels cannot be compared to determine 
which one is best, or most effective, because they are trying to do different : 
things.' Evaluation can he^p determine whether or not each of the outcomes have 
been attained for each specific program, but evaluation cannot determine which 



outcome is wosi desirable to attain or which outcome is "best." 

« Moreover, it is important to distinguish (^learly between the outcomes desired 

for a program and the process used. for attaining those/outcomes . Feiman suggests 

that there are three models of teacher centers actually in operation each char-, 

acteVized by a' specific process linked to a specifit set of desi^red outcome^. 

But. theoretically there are nine models, oae model for each combination of 

process airfid<5utcome. In addition there are a nearly endless variety of mixes 

'With some teacher centers undoubtedly using all three processes. - * 

. " » .Another Sufi^story illustrates the importance of understanding clearly the' 

difference between the processes for attaining a goal and the actual goal itself. 

Th^ incomparable Mulla Nasrudin is* visited by a would-be 
; disciple. The man, after many vicissitudes, arrives at the hut 

^ \ ^ on vthe mountain where the Mulla is sitting. Knowing that every ^ 
single action of the illuminated Sufi is meaningful, the new- c 
comer asks Na^srudin why he is blowing on his hands', /'To warm 
myself in the cold, of course.'* ^ ^ ^ . 

Shortly afterward, Nasrudin pours out ^i^bowls of soup, -^i , / 
, and blowis on his owrl. ''Why are you doing that? "Master?" asks 
the disciple. , "To cool it, of course./*s*ays the teacher. 

At this point the disciple leavfes Na%fiidin, unable^ to . * 

trust .any longer a man who uses the s^ process to arrive * 
at different results heat and cold. . 

A comprehensive evaluation will gather informatibn about both process and 

outcomes. Indeed, the processes identified by a local teacher center staff 

as those processes tiiey desire to employ Jm their work constitute a set of 

program implementation goals. Evaluates refer to the study of factors affecting 

program implementation as "process evaluation.*' Feiman's study of^he Teacher 

Curriculum Work Center .is a.p,rbcess evaluation; She studied what, teachers did 

at the Center, i.eT, t-he- processes whereby they obtain support, ideas> and 
, ^ \ ' ' ' '\ ' ' * 

materials; she did not study whether or not teachers actually^ changed what they 

did;in the class^room. Such process studies can be extremely valuable for they 
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proviW sta^f with information about whether or not the program is being inpFe 
mdntfe^ as desired. - * 



Just as it is inappropriatfe^to compare ^teachei^ 'centers that aim at different 
outcomes, it is 3lso inapp/opriate^to- cptripare the teacher cente^pproach to 
educational in-servi ce wtth other school-based or University-based app.roaches'to 
* teacher in-service. To do so is to once agaiTi search To'r The One True Way in' 
a world where different approaches work differently for ^iffererft people. ^ Each . 
approach must be empirically evaluated in its own teV^ns; the decision about whether 

.. „ ■ ■ f 

or not the goals of a particular approach are desirable is a [political question. 

From whence cometh this evaluation, preoctjxipation with ^identifying the one - 
best approach to a problem? From the earliest days of my social science graining 
I was taught that the aim of social science was to find that one set of vari'ables 
that would allow us to perfectly predict human behavior. I was taught that the 
problem of perfectly predicting human behavior was largely^ rtieasurement problem. 
If-l5e only had complete data 6n a person or group people we jC5uld predict 
with near certainty the behaviors, attitudes, and inner .workings of that person 
or' group. The equations never quite worked out biiit that was because of "measurement 
error," problems with "variable specification," and, of cours^e, the unfortunate 
^ fact that there will always be a few deviants around td^crev/ up our predidi^i^^^. 
This positi visti^: perspective has had an f^portant effec^ on government"^ 
policy-making and on research funding,* especially evaluation andWo^ icykresearch. 
; Legislators would Tike to find the- one solution, so they mandate research to te-U 
them- the answer — the or^ answer — to education,^ health, welfare, human re>dt1^ons, 

bureaucracy, Which Vducational model .is 51st? What's th'e one best way to - - ^ 

^' : organize government? Which one program approach iis most effecti've 1n reducing 
cfime and .rehabilitating criminals? / ' ^ 

" • The truth i^ that there is no 0ne best program approach across the Jpoard for 
,tHe same reason that there is no single factor or set of factors that qan solve 
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the n\ystery of human behavior, no one answer to that'tncTst fundamental philosophical 
,qu€stion: Why ao people do what they do? (And that mo^^t fundamehtal governmental 
quest'lon: How do we, get people .to do what we want them to do?) 

The reason is that individual people are different .from each other in unique' 
ways ~- and those unique 'widividual charactefisti/c4 make a differer^ce in what 
people do and -how they respond to progrants. From the structuralist perspective 
that dominates sociaJ science, individual people do not make a/difference. We 
learn in introductory sociology that the major cK^^racteristic of modeVn society 
as a rational system is the interchangeability of peopljg in positions. It doesn't 
matter who is President; the position is determined by large-scale, long-term, 
socio-polf^ic^l -economic factors that are not subject/^o personal mani|)ulation. 



It doesn't matter who^the bureaucrat is in the vf^lfai^e c^ffice; it matters oaly 




iTtioni^ Jt 



at someone, anyone fill the position/ Jt doesfl*/ matter who runs an educational 



• "program, the trick i^ to structure ;.thg^ program effectively. 

^ But it's 'just not so. Jfie predict! on^equa/jons for human behavior don't work 
perfectly because human behavior is not perfec/tly predictable. After the last of 



background variables, demographic factors, arid structural influences have been 
measured; and entered into" the equations, the accuracy of the predictions are 



downrigbt p.uny. Freedom of individual acxion, personal predilections and 
choice, unique circumstances, and the Nndamental individual!^ of human beings 
Win th^ day. 

In* educational research we keep/discovering that what maJces the' difference is 
the'enthusiasm and dedication of individual teachers, not some .theoretical model 
. of instruct^ion. In therapi^iancUcounseling what matters is the sincerity and caring 
of one individual trying to help another, not the psytiho therapeutic -method employed. 
In ^program a^ter program I ii^ that the abilities, personalities, and dedication 
of the people who run and staff programs make the maj^ difference. In effect, 
-the personal factor that I described in the first sectio^j of thfe paper as^ the - , 
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.key factor in determining the utilization of; evaluation 'research demonstrates 
the same insight. Individual people and individual circumstances, nake the major 

• • ■/ 

difference in what happens. . . , / ' 

This means that instead of searching foj^ that stngle model of educat»ion , • 

healtli, or welfare that will work everyv/here, legislators and government admin- . 

istrators must find ways to stimulate innovation and adaptation of programs to 

fit unique local needs and circumstances. Evaluation of such [5ro grams 'must also 

be based on the unique chyaci:eristics and goals of local programs ,^,not evaluation 

by standardized 'criteria applied across the board .^* 

The great threat from Big Government (or behavioral science in the service 

of Big Government) is not that it will ever perfectly, cofitrol or manipulate us. * 

The great danger; is that we must use , an ever greaterVjufe'nt^m of our personal 

energy, creativity, and freedom of action finding ways to subvert standardized 

p/escriptions for be^vior (and evaluation) that are usually irrelevant and often 

" destructive. * " ' 

The search ,for solutions will go on tojbe sure, both in science and in educa-^^ 
; •/ . 

tiorj. 'But the search ought not be for the One Solution. Rather, we must search 
for situational solutions What is h^t for this program at this time in these 
circumstances? How can we help staff accomfJlish their goals?- challenge iS 
to identify and implement alternatives not universal prescriptions that treat 
people as if they* re all the same. This places the responsibility for program 
e\^luation clearly at the local level. . . . ' 

irhis does not mean that there is no -(^esponsibi lity .fcJr^ evaluation at the 

' * • < . -J- . - ' 
t^jrational Ifevel. < Quite the contrary, the Federal responsibility is to make avail- 

able resources and incentives for the condgct^^^^^^local program evaluations. Each 

local program should describe and evaluate/both processes /what the program is 

• j \ 

doing?) and outcomes (what changes occujrin teachers and classrooms?). • These 
. local evaluations will then provide /ich tase'study data that can be content 



analyzed at the •'•national 1eve;,to look for patterns, trends, and variations in 
effett'iveness under different conditions. -^Therein will be found info?mation for 

'program planning and develop.ment* ' \^ 

\ - \ * * . 

Of course, this is not, the easy way to c^rry out evaluations. " It is hard 

on loc^il staff because they must clearly and .explicitly state objectives for both 

program processes and outcomes; and they must develop methods of describing, 

obse'rving, and measuring those processes and , outcomes. It is hard on bffice of 

Education anxJ N.I.E.. staff because they, cannot afJply some^iandy Yardstick to all 

/ • ' ' . * • . 

programs and thereby discover whi ch is best. It is* hard on evaluators because 

tney must j^nteract intensively with local staff to determine what information j« " 

appropriate in a particular situation. But if teacher centers are evaluated as ' 

other national educational programs haye been evaluated (e.g., Headstart, Follow 

Through, National Assessment), we will continue io i)e like ifasrudin looking for 

h.is lost key. ^ , i ^ 

On one occasion a neighbor found Nasrudin down on his knfees under 
a street lamp looking for soinethiTig. 

'^What have you'f^st, Mulla?" ) ^ ^ 

^ ^ * "^1y key," said Nasrudin. > - 

After a few minu-tes of searching, the other man said, "Where did 
you drop it?" jX^ ^ , * . ..^ 

"I^n that dar^pasiture." ' ( ' • ^ 

*"Then why, fbr heaven's sake, are you booking here?" 

"Because there is more light here*." 



On Causes^ and Effects 



/ 



The reason for separating p/ogram^ processes from program outcomes is so that 
they can be related to each other. The sequence of questions goes like this: 

T. Are we implementing the program the way we want to (process 
evaluation)?^ 

2. Are teachers changing/developing the way w^ want^them-to 
(outcomes evaluation)? 
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'3. Is what we're doing in the program (the processes) reUt9<J to 
^. - .the changes we. see in teachers (th^outconfes)? • ' • * 

% / * * 

Thi§ last question "tak^s us into the arena of causality . ^Do the prdc-esses/ * 

* I " ' 

activities of the teacher center cause/affect the beha\/iorsyatti tudes/slTi^lls/ 

know! edge/ feelings of teachers? One need- know very little about research'^to know 

Cat it is impossible to establish causality in any final sense when dealing with 
e complexities of real programs where. tVeatmehts ^re never purei single^ and 
unco'ntamin'ated. - * • . ' 

It is'easy to become frustrated with^ the di;(^ficulty of establ ishing .thq ^ 
relationsHIp betwjeen program activities and program outcomes. We can't answer" 
such questions definitively. No amount of scientific sophistication will provide 
defini ti-^e Answers to causal ques'tions fo^t^'^^^&k same reason that there is 'no 
' One rigflt ^approach to changing human behavi&r. But that is no reason not to ask 

the q-uestion. We cannot proviNJe "definitive answers but we can arrtye at some 
" reasonafetle estimation of the litcelihood that particular activities^ have had 'an effect 
. This ^oint^is important for it tells us something about w|at we can expect. 



.from evaluation research. Evaluation data are nevfer clear-rtjut and a^b^olute.- Studies 
are always flawed in some way. There are alwa(ys questions.of reliability and ^ « 
va^lidity. Error-free instruments do not exist and cannot exist in the measurement 
of complex human social/behayioral/psychaJpgi cal phenomena,. Of what good is all 
•This then? • . ^ ^ * , 

"^vi^lii^tioit research is only of use if you believe thafsome sys^^atic infor- 
mation'is better than n(<^systemat*ic information. Evaluation research only has 
meaning if you believe tha* a rough idea of the relationship betweea program activ- 
ities and* outcomes is better than reliance (entirely upon hope and good intenticlns. 
Evaluation .research does not provide final answers but It can p-rovide direction . 
' Earlier I referred to a s-tudy aimed at determining^lhe factors .that affect 
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the utilization of evaluation research. The first task in that study was to 
describe how evaluations actually were used. What we found was considerably "more 
complex an^^l^dism^l ' than our original impressions led us to expect. Evaluation 
" research is used but not in the ways we had anticipated. None of the impacts 
described was of the type where new findings from an eval uation. led, directly and 
immediately to the making, of major, concrete pro-am decisions. The more typical 
.Impact was one where the eval uation' findiags provided additional pieces of infor- 
mation in the difficult puzzle of program action, thereby permitting some reduction 
in the uncertainty, within which any decisionmaker inevitably operates, ^^C^ 

This reduction of uncertainty emerged, as highly important to decisionmakers. 
In some cases it simply made them more confident and determined. On the other 
hand, v/here the need for cfiange was indicated, .an evaluation study couljl help 
speed up the process of change or provide an impetus for finally getting things 
rolling, • 

Reducing uncertainty, speeding things up, and gett^.ng things finally started 

are real impacts not'revolutionary, organization-shaking impacts but important 

• impacts in the opinion of the people we interviewed. One a,dminis trator summarized 

this view both on the, specific evaluation in question and .about evaluation in' ' 

general : . . ' > • 

M/self I have a favorable view toward evaluating. If nothing else 
it precipitates activity many times that could not be precipitated 
• ^ wifthout someone taking a hard look at an organi Aition, It did 
precipitate activity in [this program] Some of the findings were 
not positive,- Some of .it was negative. At least something occurred 
» that wouldn't have occurred if the evaluation hadn't, taken place, 

It. is crucial to set realistic e^cpectations at the beginning of the evaluative 

process about what' can be expected arti' how data will be used. Everyone involved 

should clearly understand that evaluation findings can provide important information 
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for program improvement, but findings must be interpreted in the full context of-, 

The particular circumstances under which a program operates. Evaluation results 

from a single study ought not be the sole basis on which major pro gram^decis ions 

are made. t* 

It is important, then, to ask caus'al. questions , even though evaluation data 

can only provide an approximate picture of what is really happening. It is 

important to ask causal questions. ' It is also important to interpret the results 

with prudence and care. 

One day an old man approached Zen master HyakujoT. The 
^ old man/s^iid, "I am not a human being. In ancient times I lived 

on this mountain. A student of the Way ai^sked me if the enlightened 
were still affected by causality. /l replied saying that they were 
not affected. Becuase of that I was degraded to lead the life of 
a wild fox for five hundred years. I now request you to answer * 
one thing -for me. Are tfie enlightened still affected by causality?"* 

Master Hyakujo replied, "They are not deluded by causality." 

At. that the old man was enlightened. 

Mastei^^yakgjQ clapped his hands, laughed, and said, "I only 
thought that the barbarian *s , beard was red, I never realized it 
was a red-bearded barbarian. "25 ' , 



QUESTIONS FOR TEACHER 'CENTER EVALUATIOI^ 
• ' AN OVERVIEW FROM AN IRON COW' 

I began this paper with the basic tenet df the Koan^.educitibnal method that 

the question is as important as the. answer*. This paper has centered on evaluation 

questions for teacher centers. In this final sectidn I shall- attempt to summarize 

and order those questions.' (' ' 

*^ 5 - ' ■ V 

, 1. Who is the study for?^Whose question will be answered? 

Who wi IV determine what it all means? Begin by .contem- 
• plating the usefulness of what to whom. , . ' 

a. ^ Is the evaluation for local program staff to use 

' ' in improving the program? . v 

b. Is the evaluation for state or federal officials, to 
use in making global program decisions? 
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c. What di^ectations" are there about how the evaluation 
data wi ll be used, by whom, to make what decisions^ 
on what issues? 

VJhat do we want to a/fect in teachers? How do we want teachers* 
to be different after the* program? 

a. What changes, if ^ny, do we expect in- teachers ' feelings? 

b. " What. changes, if any, do we expect in teachers' opinions? \ 

c. What changes J if any , do we expect in teachefs ^ knowledge? 

d. What changes, if any, do we expect in teachers' skills ? 

'e. What changes, if any, do we expec,t in teachers' behavior? ^ 

How will we observe, describe, and/or measure the degree to which' 
teachers are different after the program? 



What are our criteria for success? and failure? At what level 

and to what degree do we expect to affect teachers (for ea'ch-specffi 

outcome desired)? * ^ . . 

a. At what level are we doing an outstanding job? - 

b. At what level are we doing an adequate job? 

c. At wlrat level are we doing a poor job? 

What ^program activities and processes do we^expect to engage in 
to affect teachers? - ' • ^ 

a. Do^we expect to use formal .instruction by curriculum 
specialism's to affect teachers? 

b. Do we expect to u$e informal, non-directed teacher 
exploration to affect teachers?* 

c. Do we exf^ect to use advisors in establishing warm, 
interpersonal, and directive relationships with teachers?* 

How will we oSserv^ dfecrit^e, and/Or measure the degree to which 
We actually implement program processes and activities as planned? 



What are our criteria fgr success? ^ failure? At what^level 
and to what degree do we expect to* implement program processes 
and activities? / . ^ ^. ' 

a. At what level are we doing an outstanding job? ^ 

b. At what level are we doing an adequate job? • * 
. c. _At v/hat ♦level are we doing a poor job? 



What is the relationship between what we're doing in 
the program (the processes) and the changes observed in 
teachers (the. outcomes)? Is there a'relationshi p? \^ 

a. How mtich confidence do we have in the strength, of 
^that relationship? 

What's it all mean? What do we do? 'What have we learned 
about the program that gives us a direction for action? 



10. The Koan of An Iron Cow . . - , 

' Master Fuketsu said, "Buddha-mind is'-just like an iron cow; 
if there is niovement--there is no progress; if there is ^ 
standstill--there is stagnation. ^ Well , this ^no-movement- 
no-standstilj , * should one be mindful of it? Should one be 
unmindful of it? 



Answer: It's, like a stone mill.* 
Master: Why is it so? • » ^ 



\ 



Answer; It do€5^n'.t mgve a bit." Along the hedge chasing a 
butterfly, -by the water's side toying with a frog. 

Master: Jlm^^.lno-'maverffeht-^^^ , ' should one 'be 

mindful of it? Should one be unmindful of it? 
If I keep after yoJ in this way, how .will you 
answer? 



Answer: The pupil slaps his master. once. 
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